Repository navigation
fix(review): verdict trust: only the engine blocks, policy from the base branch, allowForceTier - #1722
fix(review): verdict trust: only the engine blocks, policy from the base branch, allowForceTier#1722RaghwendraSingh wants to merge 3 commits into
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
|
Thanks for your contribution! This PR doesn't have a linked issue. All PRs must reference an existing issue. Please:
See CONTRIBUTING.md for details. |
There was a problem hiding this comment.
4 issues found across 12 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/opencode/test/altimate/review.test.ts">
<violation number="1" location="packages/opencode/test/altimate/review.test.ts:145">
P2: This fixture update leaves the unknown-confidence regression test ineffective: `u()` has no engine evidence, so its findings are classified as catalog and excluded before confidence is evaluated. Add deterministic-engine evidence to `u()` (or reuse the engine fixture) so the test proves three unknown engine warnings cannot block.</violation>
</file>
<file name="packages/opencode/src/altimate/review/format.ts">
<violation number="1" location="packages/opencode/src/altimate/review/format.ts:221">
P2: This note misidentifies the policy file when the repository uses the supported `.altimate/review.yaml` or `.altimate/review.json` filename. Use a generic `review policy` label or carry the selected filename into the envelope before rendering.</violation>
</file>
<file name="packages/opencode/test/altimate/review-verdict-trust.test.ts">
<violation number="1" location="packages/opencode/test/altimate/review-verdict-trust.test.ts:115">
P3: These integration tests leak one temporary Git repository per case. Use the repository test fixture or remove each root in a `finally` block so repeated test runs do not accumulate temp directories.</violation>
</file>
<file name="packages/opencode/src/altimate/review/finding.ts">
<violation number="1" location="packages/opencode/src/altimate/review/finding.ts:107">
P1: The namespace prefix is too broad for the trust boundary: an unrecognized `altimate_core.*` tool is promoted to `engine` and can block. Replace the prefix check with an explicit allowlist of the known deterministic evidence tools so new tools remain catalog until classified.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| if (f.origin) return f.origin | ||
| const tool = f.evidence?.tool ?? "" | ||
| if (tool === "ai-review") return "ai" | ||
| if (tool.startsWith("altimate_core.") || ENGINE_TOOLS.has(tool)) return "engine" |
There was a problem hiding this comment.
P1: The namespace prefix is too broad for the trust boundary: an unrecognized altimate_core.* tool is promoted to engine and can block. Replace the prefix check with an explicit allowlist of the known deterministic evidence tools so new tools remain catalog until classified.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/finding.ts, line 107:
<comment>The namespace prefix is too broad for the trust boundary: an unrecognized `altimate_core.*` tool is promoted to `engine` and can block. Replace the prefix check with an explicit allowlist of the known deterministic evidence tools so new tools remain catalog until classified.</comment>
<file context>
@@ -84,9 +84,36 @@ export const Finding = z.object({
+ if (f.origin) return f.origin
+ const tool = f.evidence?.tool ?? ""
+ if (tool === "ai-review") return "ai"
+ if (tool.startsWith("altimate_core.") || ENGINE_TOOLS.has(tool)) return "engine"
+ return "catalog"
+}
</file context>
There was a problem hiding this comment.
Follow-up: low priority, tracked. Switch to an explicit allowlist of engine tools.
| notes.push(`${env.scope.outOfScopeFiles} changed file(s) outside the dbt project were not reviewed.`) | ||
| if (env.policy?.changedInPr?.length) { | ||
| notes.push( | ||
| `This PR changes the review policy (\`.altimate/review.yml\`: ${env.policy.changedInPr.join(", ")}). ` + |
There was a problem hiding this comment.
P2: This note misidentifies the policy file when the repository uses the supported .altimate/review.yaml or .altimate/review.json filename. Use a generic review policy label or carry the selected filename into the envelope before rendering.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/format.ts, line 221:
<comment>This note misidentifies the policy file when the repository uses the supported `.altimate/review.yaml` or `.altimate/review.json` filename. Use a generic `review policy` label or carry the selected filename into the envelope before rendering.</comment>
<file context>
@@ -216,6 +216,12 @@ export function coverageNotes(env: VerdictEnvelope): string[] {
notes.push(`${env.scope.outOfScopeFiles} changed file(s) outside the dbt project were not reviewed.`)
+ if (env.policy?.changedInPr?.length) {
+ notes.push(
+ `This PR changes the review policy (\`.altimate/review.yml\`: ${env.policy.changedInPr.join(", ")}). ` +
+ "The base branch's policy was applied; the change takes effect after merge.",
+ )
</file context>
| `This PR changes the review policy (\`.altimate/review.yml\`: ${env.policy.changedInPr.join(", ")}). ` + | |
| `This PR changes the review policy (${env.policy.changedInPr.join(", ")}). ` + |
There was a problem hiding this comment.
Follow-up: low priority, tracked. Say "review policy" generically.
| }) | ||
|
|
||
| async function repo(baseYml: string | undefined, headYml: string | undefined) { | ||
| const root = await fs.realpath(await fs.mkdtemp(path.join(tmpdir(), "review-policy-"))) |
There was a problem hiding this comment.
P3: These integration tests leak one temporary Git repository per case. Use the repository test fixture or remove each root in a finally block so repeated test runs do not accumulate temp directories.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/test/altimate/review-verdict-trust.test.ts, line 115:
<comment>These integration tests leak one temporary Git repository per case. Use the repository test fixture or remove each root in a `finally` block so repeated test runs do not accumulate temp directories.</comment>
<file context>
@@ -0,0 +1,174 @@
+ })
+
+ async function repo(baseYml: string | undefined, headYml: string | undefined) {
+ const root = await fs.realpath(await fs.mkdtemp(path.join(tmpdir(), "review-policy-")))
+ await g(root, "init", "-q", "-b", "main")
+ await g(root, "config", "user.email", "t@test")
</file context>
There was a problem hiding this comment.
Not changing in this PR: test-hygiene nit.
|
|
||
| /** File content at a commit (`git show <ref>:<path>`), or undefined when the | ||
| * path does not exist there. `rel` is repository-relative, POSIX. */ | ||
| export async function showAtRef(cwd: string, ref: string, rel: string): Promise<string | undefined> { |
There was a problem hiding this comment.
WARNING: A failed git show is silently treated as "no config at the base commit"
readFirst treats the undefined returned here as "candidate absent" and moves on, so loadReviewPolicy falls back to DEFAULT_REVIEW_CONFIG (mode: comment, allowForceTier: true) and stamps policy.source: "default" — indistinguishable from "the base branch has no config". Any git-level failure (bad object, missing shallow history, permissions, git erroring for a reason other than the path being absent) therefore silently demotes a gate policy to the permissive default instead of surfacing it, which is the opposite of the trust property this PR adds. git cat-file -e <ref>:<path> (or checking the exit code/stderr) would distinguish "path missing at that commit" from "git failed" and let the latter throw.
There was a problem hiding this comment.
Fixed in 956f4ef (see the cubic thread on showAtRef).
| if (origin === "engine") return true | ||
| if (origin === "ai") return false | ||
| const rule = catalogRuleId(f) | ||
| return !!rule && rubric.blockingRules.includes(rule) |
There was a problem hiding this comment.
WARNING: blockingRules entries are matched by exact string and never validated
Rule ids are emitted by two catalogs with different conventions (rule-catalog uses hardcoded-credential, dbt-patterns uses pii_into_mart), and nothing checks that a configured id exists. A typo, a case difference, or the other convention therefore leaves the gate silently non-blocking — the exact failure mode a team opting a rule in wants to avoid — with no error and no note in the comment. Consider validating rubric.blockingRules against the known catalog ids when the rubric is resolved (warning when an entry matches nothing) and documenting where teams can find the ids.
There was a problem hiding this comment.
Follow-up: low priority, tracked. Warn on ids that match no catalog rule.
| changedInPr: string[] | ||
| } | ||
|
|
||
| function canonical(value: unknown): string { |
There was a problem hiding this comment.
SUGGESTION: canonical duplicates stableStringify in verdict.ts
verdict.ts:278 already implements this exact algorithm (keys sorted at every depth, array order preserved, undefined dropped) for the signature body. Export that helper (or move it to a shared module) and use it for policyHash too: policy.hash is signed alongside the signature body, so the two canonicalizations should stay identical by construction instead of being two copies that can drift.
There was a problem hiding this comment.
Not changing: nit. They serve different modules; consolidation can happen in a cleanup pass.
| }): Promise<ReviewPolicy> { | ||
| if (!opts.readBase) { | ||
| const config = await loadReviewConfig(opts.cwd) | ||
| const present = await readFirst((rel) => fs.readFile(path.join(opts.cwd, rel), "utf8").catch(() => undefined)) |
There was a problem hiding this comment.
SUGGESTION: The no-git branch reads the same candidate file twice
loadReviewConfig above has already scanned CANDIDATE_FILES and read the first existing file; this second readFirst repeats the same read sequence only to learn whether a file was present, and the candidate loop is now written a third time here. Folding both into one helper that returns { text, rel } | undefined (and parsing that text once) removes the extra I/O and the chance of the two passes disagreeing if the file changes between them.
There was a problem hiding this comment.
Not changing: nit, one extra small file read on a non-default path.
| const out: string[] = [] | ||
| const keys = new Set([...Object.keys(base), ...Object.keys(head)]) as Set<keyof ReviewConfig> | ||
| for (const k of [...keys].sort()) { | ||
| if (canonical(base[k]) === canonical(head[k])) continue |
There was a problem hiding this comment.
SUGGESTION: Comparing parsed partial configs reports settings spelled out at their default values as changes
Representation, not effective policy, drives this comparison. A base file with mode: gate parses to rubric: {}; a PR that adds rubric:\n thresholds:\n warehouseCostMinRows: 1000000 (the default) parses to a full rubric.thresholds object, so the PR comment claims it "changes the review policy (rubric.thresholds)" that "takes effect after merge" even though the resolved rubric is identical. Comparing resolveRubric(...) results (and normalized top-level values) instead of the raw partials would keep the note to genuine behavior changes.
There was a problem hiding this comment.
Follow-up: low priority, tracked. Compare resolveRubric() outputs.
| throw new Error("--force-tier is disabled by `allowForceTier: false` in the base branch's .altimate/review.yml") | ||
| } | ||
| if (opts.manifestPath) config.manifestPath = opts.manifestPath | ||
| if (opts.mode) config.mode = opts.mode |
There was a problem hiding this comment.
SUGGESTION: policy.hash is computed before these overrides, so it can describe a different policy than the envelope
loadReviewPolicy hashes the file config, then this line and the severityThreshold override mutate config, and the envelope later records policy.hash next to mode: config.mode / severityThreshold. With --mode gate (which github/review/action.yml always passes) the signed hash says comment while the signed mode says gate, so an auditor cannot check that the hashed policy is the one that was applied. Hashing after the overrides, or documenting that the hash covers only the config file, would remove the ambiguity.
There was a problem hiding this comment.
Follow-up: low priority, tracked (same as the cubic thread).
| title: "t", | ||
| body: "b", | ||
| file: `m_${Math.random()}.sql`, | ||
| evidence: { tool: "altimate_core.check", result: {} }, |
There was a problem hiding this comment.
SUGGESTION: This makes the unknown-confidence test pass for the wrong reason
Adding engine evidence to mk is right for the blocking assertions, but "unknown-confidence warnings do NOT accumulate into a block" (u() around line 168) builds findings with raw makeFinding and no evidence. Those are now catalog origin and ineligible regardless of confidence, so the f.confidence !== "unknown" clause in computeIdealVerdict is no longer exercised by that test. Giving u() engine evidence (altimate_core.equivalence) keeps it guarding the confidence rule.
There was a problem hiding this comment.
Follow-up: low priority, tracked (same as the cubic thread).
| ### Which policy applies, and what can block | ||
|
|
||
| - **The base branch's `.altimate/review.yml` applies.** The file is read from the | ||
| commit the PR branched from, so a PR cannot relax the rules it is reviewed |
There was a problem hiding this comment.
SUGGESTION: The docs state a stronger trust boundary than the shipped path enforces
The policy file is base-scoped, but github/review/action.yml always passes --mode/--severity from the workflow (args=(--mode "$IN_MODE" ...)), and on pull_request events that workflow comes from the PR. A PR can therefore still switch the gate to comment mode by editing .github/workflows/*.yml — the PR description notes this as out of scope, but this bullet reads as if no PR-side edit can relax the gate. Adding that caveat (and pointing at CODEOWNERS / branch protection on .github/) would keep the doc accurate.
There was a problem hiding this comment.
Fixed in 956f4ef. Workflow flags can no longer relax a base policy, and the docs state the remaining caveat (protect .github/).
Code Review SummaryStatus: 5 Issues Found | Recommendation: Address before merge Overview
Review scope: merge-base Issue Details (click to expand)SUGGESTION
The four carried suggestions are on lines outside the PR's hunk-level additions and already have active inline comments, so they are summary-only here. Files Reviewed (12 files)
Notes / assumptions: this is a read-only review, so tests were not executed; the test-quality finding is derived from reading the assertions. Known residual gaps are documented in the PR and the docs (a caller that supplies both Fix these issues in Kilo Cloud Previous Review Summary (commit 8ce9cec)Current summary above is authoritative. Previous snapshots are kept for context only. Previous review (commit 8ce9cec)Status: 8 Issues Found | Recommendation: Address before merge Overview
The design is sound and the core claims check out: Issue Details (click to expand)WARNING
SUGGESTION
Notes / assumptions: this review found no path where the new code makes a non-engine finding block, and the AI lane remains non-blocking. The documented residual gap (pre-supplied Files Reviewed (12 files)
Reviewed by deepseek-v4.1-flash-1017 · Input: 336.4K · Output: 81.6K · Cached: 7.3M Review guidance: REVIEW.md from base branch |
8ce9cec to
4004ec6
Compare
|
Kilo Code Review could not run — your account is out of credits. Add credits or switch to a free model to enable reviews on this change. |
hkaltai
left a comment
There was a problem hiding this comment.
Verdict: approve (no CRITICAL/MAJOR). Panel: Claude, Codex, lead. The design is sound: unclassified tools default to catalog, so new detectors cannot block until classified; the AI lane stays non-blocking; the policy is read from the merge-base and signed in the envelope. I could not run the Bun suite in the review worktree (dependencies not installed); findings were verified by reading the code and isolated probes.
Counts: CRITICAL 0, MAJOR 0, MINOR 6, NIT 6.
Also worth a changelog/release note: pattern-rule criticals (cross_join, pii_into_mart, etc.) and non-engine warning counts stop blocking by default, and local edits to review.yml no longer apply to the PR being reviewed. Missing tests: hyphen/underscore blockingRules ids, hash after CLI overrides, the no-git path (changedFiles + getContent), a cwd in a repo subdirectory, base-read failure and an invalid base file.
Nits
finding.tsENGINE_TOOLS (~L96):altimate_core.lexicalis a pre-parse lexical check andschema.detect_piiis text-based, yet both count as "engine"; the comment "(AST, manifest, lineage, warehouse)" overstates it. Fix: adjust the comment or classify them explicitly.config.tscanonical()(~L136): duplicatesstableStringifyinverdict.ts. Fix: export and reuse it.verdict.tscomputeIdealVerdictdoc: "the docs promise that only the deterministic engine blocks" references docs instead of stating the invariant. Fix: state the invariant directly.finding.tscatalogRuleId: also returns ids for engine findings. Fix: rename toruleIdOf.rubric.tsblockingRules: never affects engine findings. Fix: considerblockingCatalogRules.format.ts(~L24-36): a non-blocking catalog critical renders as "Reviewed with comments, 1 critical" with no hint why it did not block. Fix: add a footnote when origin filtering lowered the verdict.
| if (origin === "engine") return true | ||
| if (origin === "ai") return false | ||
| const rule = catalogRuleId(f) | ||
| return !!rule && rubric.blockingRules.includes(rule) |
There was a problem hiding this comment.
[MINOR] blockingRules matches ids by exact string; catalog sources spell ids differently
return !!rule && rubric.blockingRules.includes(rule)rule-catalog ids are hyphenated (hardcoded-credential) while dbt-patterns ids use underscores (pii_into_mart, cross_join); orchestrate.ts already normalizes _ to - when comparing ids. Failure: blockingRules: ["pii-into-mart"] yields COMMENT, ["pii_into_mart"] yields REQUEST_CHANGES, so a team opting a rule in with the other spelling silently gets no blocking.
Fix: compare after normalizing _ to - on both sides, and list valid ids in the docs or warn on unknown ids at config load. (Claude, lead; premise-checked)
There was a problem hiding this comment.
Good catch, thanks. Fixed in fix(review): address review feedback: blockingRules spelling, origin from evidence only, hash after flags, base-policy error. Ids compare through normalizeRuleId (lowercase, _ as -) on both sides, so pii-into-mart and pii_into_mart both opt the rule in. Tested. Warning on unknown ids at config load is still a follow-up.
| const cwdRel = await repoRelative(gitRoot, opts.cwd, true) | ||
| const policy = await loadReviewPolicy({ | ||
| cwd: opts.cwd, | ||
| readBase: |
There was a problem hiding this comment.
[MINOR] Callers supplying changedFiles + getContent get no base-policy protection
readBase:
resolved && cwdRel !== undefinedWhen both are supplied, needGit is false, resolved is undefined and the working-tree policy applies, even if base was passed explicitly. The PR description states this limit, so I rate it MINOR (Codex rated MAJOR; downgraded since it is documented and the git-less path has no trusted base to read). Scenario: an integration with pre-computed diffs reviews a PR whose review.yml sets mode: comment and it is honored.
Fix: let such integrations supply a trusted readBase/policy text, and emit a visible note when source is working_tree in a PR context. (Codex)
There was a problem hiding this comment.
Thanks. I'm not changing this in this PR. That path exists for integrations without git history, so there is no trusted base to read, and the CLI and the action always resolve git. It is documented in the PR description. A caller-supplied readBase and a visible note for source: working_tree in a PR context are the right follow-up, and they are tracked.
| throw new Error("--force-tier is disabled by `allowForceTier: false` in the base branch's .altimate/review.yml") | ||
| } | ||
| if (opts.manifestPath) config.manifestPath = opts.manifestPath | ||
| if (opts.mode) config.mode = opts.mode |
There was a problem hiding this comment.
[MINOR] policy.hash is computed before CLI overrides
if (opts.mode) config.mode = opts.modeThe hash is taken inside loadReviewPolicy, then --mode/--severity/--manifest mutate the same config. envelope.policy.hash therefore describes the base file, not the policy actually applied (mode is signed separately, but severityThreshold and manifestPath overrides are recorded nowhere). Repro: base mode: gate hash differs from the hash of the post---mode comment config.
Fix: recompute the hash after overrides, or record an overrides list beside it, or document the hash as the base-file policy. (Claude, Codex)
There was a problem hiding this comment.
Agreed, thanks. Fixed in the same commit: the hash is recomputed after --mode/--severity/--manifest, so it describes the applied policy. There's a test that a tightened run has a different hash.
| * path does not exist there. `rel` is repository-relative, POSIX. */ | ||
| export async function showAtRef(cwd: string, ref: string, rel: string): Promise<string | undefined> { | ||
| try { | ||
| return await git(["show", `${ref}:${rel}`], cwd) |
There was a problem hiding this comment.
[MINOR] showAtRef treats every git error as "file absent"
return await git(["show", `${ref}:${rel}`], cwd)A failure other than a missing path (missing object in a partial clone, permissions) silently yields default policy (source: "default"), dropping a strict base policy with no message.
Fix: return undefined only for git's "does not exist in" / "exists on disk, but not in" errors; rethrow otherwise. (Claude)
There was a problem hiding this comment.
Thanks. Fixed in fix(review): address review comments: workflow flags cannot relax base policy, showAtRef fails closed: only git's "does not exist in" / "exists on disk, but not in" map to absent, and anything else is rethrown. Tests cover a missing path, a present path and a bad ref.
| try { | ||
| config = baseText === undefined ? structuredClone(DEFAULT_REVIEW_CONFIG) : parseReviewConfig(baseText) | ||
| } catch (e: any) { | ||
| throw new Error(`Failed to load the base branch's .altimate/review config: ${e?.message ?? e}`) |
There was a problem hiding this comment.
[MINOR] An unparsable base review.yml blocks the PR that would fix it
throw new Error(`Failed to load the base branch's .altimate/review config: ${e?.message ?? e}`)If the base file is invalid, every review throws, including the fix PR, so recovery needs an admin bypass. Previously the fix PR could repair itself. Failing closed is defensible; consider a clearer message naming the recovery path, or falling back to the PR's parsable version with a loud note.
(Claude)
There was a problem hiding this comment.
Fair point, thanks. Failing closed stays, so a PR can never relax its own policy by breaking the base file. In the same commit the error now names the recovery path: fix the file on the base branch, or merge the repair PR with an admin bypass. Tested.
| /** The layer a finding came from. Unknown or missing evidence counts as | ||
| * `catalog`, so a new detector can never block until it is classified. */ | ||
| export function findingOrigin(f: Pick<Finding, "evidence" | "origin">): FindingOrigin { | ||
| if (f.origin) return f.origin |
There was a problem hiding this comment.
[MINOR] An explicit origin overrides evidence.tool
if (f.origin) return f.originThe PR says origin is derived from evidence.tool, but a pre-set value wins. Codex showed three ai-review warnings carrying origin: "engine" yielding REQUEST_CHANGES. No in-repo producer sets origin (ai-review builds findings field by field), so this is only reachable via externally supplied findings.
Fix: always derive from evidence, or reject mismatched supplied origins. (Codex; premise-checked)
There was a problem hiding this comment.
Agreed, thanks. Fixed in the same commit: findingOrigin always derives from evidence.tool, and buildEnvelope re-derives it for every finding. Three AI warnings with origin: "engine" now give COMMENT (tested).
…ase branch, `allowForceTier` - Findings carry `origin: engine | catalog | ai` (from `evidence.tool`; unknown tools count as catalog). Only engine findings count toward REQUEST_CHANGES, matching the documented promise; a pattern rule can be opted in by id via `rubric.blockingRules`. The AI lane never blocks. - `.altimate/review.yml` is read from the merge-base commit, so a PR cannot relax the rules it is judged by. Settings the PR changes are listed in the comment and envelope (`policy.changedInPr`) and take effect after merge. The applied policy's source and hash are signed into the envelope. - `allowForceTier: false` in the base policy rejects `--force-tier` (forced runs were already recorded as `tier_forced` in `review_run`). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e policy, `showAtRef` fails closed - `--mode` / `--severity` (always passed by the action from the PR-editable workflow) can only tighten a base-branch policy: `mode: gate` is kept and a higher severity threshold is ignored, each with a stderr notice. Docs state this and recommend protecting `.github/` with CODEOWNERS. - `showAtRef` returns undefined only when the path is absent at the ref and rethrows any other git failure, so the base-policy loader no longer falls back to the permissive default on a bad ref or git error. - Tests for both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n from evidence only, hash after flags, base-policy error - `verdictEligible`: `blockingRules` ids compare after `normalizeRuleId` (lowercase, `_` as `-`), so `pii-into-mart` and `pii_into_mart` both opt the rule in. - `findingOrigin` always derives from `evidence.tool` and `buildEnvelope` always re-derives it: a supplied `origin` cannot make an AI or catalog finding block. - `reviewPullRequest`: the signed `policy.hash` is recomputed after the `--mode`/`--severity`/`--manifest` flags, so it describes the applied policy. - An unparsable base `review.yml` still fails closed; the error now names the recovery path. - Tests: both id spellings, a supplied `origin: "engine"` on AI findings, hash after flags, unparsable base policy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1c5dc3a to
4726770
Compare
4004ec6 to
b354b9f
Compare
|
@hkaltai, re your review: Thanks for the approval. One commit on top:
|
| test("--mode and --severity from the workflow cannot relax a base policy, only tighten it", async () => { | ||
| const root = await repo("mode: gate\nseverityThreshold: warning\n", undefined) | ||
| const relaxed = await reviewPullRequest({ cwd: root, head: "HEAD", noAi: true, mode: "comment", severityThreshold: "critical" }) | ||
| expect(relaxed.mode).toBe("gate") |
There was a problem hiding this comment.
SUGGESTION: The --severity half of this test is never asserted
The test is named for both --mode and --severity, and line 188 passes severityThreshold: critical against a base policy of warning — the exact vector where a PR-editable workflow could hide warnings. But the only assertion is on mode, so the clamp at run.ts (SEVERITY_ORDER[opts.severityThreshold] > SEVERITY_ORDER[config.severityThreshold]) is unverified: it could be changed to always apply the flag and this test would still pass. Either build the fixture so the head change yields a warning finding and assert it survives --severity critical, or record the applied threshold in the signed envelope (policy) and assert it there — the latter would also make the effective threshold auditable, which today only the opaque hash covers.
Changes since review: Review feedback commit:
blockingRulesspelling, finding origin taken from evidence only, the policy hash computed after workflow flags, and a clear error when the base policy can't be read.Issue for this PR
Closes #1792
Stacked on #1721, which is stacked on #1720 and #1719; merge in that order.
Type of change
What does this PR do?
The README and the docs promise "only the deterministic engine blocks". In the code, any three non-AI warnings, or one regex-catalog critical (
join-no-on,create-or-replace,pii_into_mart, …), produced REQUEST_CHANGES. Separately,.altimate/review.ymlwas read from the PR's own checkout. So a PR could setmode: comment, drop ablockOncategory,excludeits own paths, or narrowreviewers, and pass under its own relaxed rules.origin: engine | catalog | ai, derived fromevidence.tool. Unknown tools count ascatalog, so a new detector can't block until someone classifies it. Onlyenginefindings count toward REQUEST_CHANGES. A pattern rule a team trusts can be allowed to block viarubric.blockingRules: [<rule id>].originis stamped inbuildEnvelopeand covered by the signature..altimate/review.ymlis read at the merge-base commit (git show <base>:<path>). The PR's version is parsed only to list what it changes; those settings show in the comment and inpolicy.changedInPrand apply once merged. The envelope signspolicy.sourceandpolicy.hash. Callers without git (pre-supplied files and content) still read the working tree.--force-tier:allowForceTier: falsein the base policy rejects the flag. Forced runs were already recorded astier_forcedinreview_run.Behavior changes:
review.ymlno longer apply to that PR.pii_into_martregex-twin tests now expect COMMENT, and the verdict unit tests use engine evidence.Not covered here: CLI flags in the workflow file (
--mode,--severity) still override the policy. Onpull_requestevents GitHub runs the workflow from the PR, so a PR could edit those flags too. Branch protection on.github/(CODEOWNERS) is the usual control there.How did you verify your code works?
test/altimate/review-verdict-trust.test.ts(11 tests):blockingRulesopt-in, and engine criticals and warning patterns still blockingGITHUB_BASE_REF: a PR relaxingmode/exclude/aistill judged in gate mode with the change shown; a PR addingreview.ymlnot applied; an unchanged policy applied with no note;allowForceTier: falserejecting--force-tierbun typecheckis clean.Cost side effects
None. One extra local
git showper run.Screenshots / recordings
N/A (CLI).
Checklist
🤖 Generated with Claude Code
Summary by cubic
Makes the review verdict deterministic and tamper-proof: only the deterministic engine can block, and a PR can no longer relax the policy it is reviewed under.
Old behavior: any three non-AI warnings, or one regex-catalog critical, produced REQUEST_CHANGES, and
.altimate/review.ymlwas read from the PR's own checkout. Now pattern-rule criticals and warning counts only comment unlessrubric.blockingRulesopts a catalog rule in, and the policy comes from the merge-base commit so a PR's edits are listed but only take effect after merge.allowForceTier: falsein the base policy rejects--force-tier.Behavior changes
origin(engine|catalog|ai) derived fromevidence.tool; a suppliedoriginis ignored, and unknown tools count ascatalog, so a new detector can't block until classified. Origin is stamped in the envelope and covered by the signature.enginefindings count toward REQUEST_CHANGES; the AI lane never blocks.blockingRulesmatches rule ids with either-or_spelling.review.ymlshow in the comment andpolicy.changedInPr, apply after merge, and the applied policy's source and hash (after flag overrides) are signed into the envelope. Non-git callers still read the working tree; an unparsable base file fails closed.--modeand--severityflags can only tighten a base policy: they can't turnmode: gateintocommentor raise its severity threshold.Written for commit b354b9f. Summary will update on new commits.